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1.0  Introduction 

The  United  States  Air  Force  (USAF)  is  in  the  process  of  institutionalizing  Human  Systems  Integration  (HSI) 
as  an  embedded  process,  to  ensure  human  considerations  and  performance  capabilities  and  limitations 
are  addressed  in  the  life  cycle  management  of  USAF  weapons  systems.  Interim  Department  of  Defense 
Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition  System,  states,  "The  Program  Manager 
will  plan  for  and  implement  human  systems  integration  (HSI)  beginning  early  in  the  acquisition  process 
and  throughout  the  product  life  cycle.  The  goal  will  be  to  optimize  total  system  performance  and  total 
ownership  costs,  while  ensuring  that  the  system  is  designed,  operated,  and  maintained  to  effectively 
provide  the  user  with  the  ability  to  complete  their  mission." 

Regardless  of  the  sophistication  of  AF  systems,  optimized  total  system  performance  is  contingent  upon 
the  warfighter's  ability  to  use  systems  fully  and  effectively  to  accomplish  the  mission.  HSI  provides  an 
integrated  approach  to  considering  the  human  on  par  with  the  hardware  and  software  they  are 
expected  to  operate,  maintain,  and  support.  Effective  management  of  HSI  design  considerations  and 
tradeoffs  within  overall  system  engineering  and  program  management  is  essential  to  achieving  the 
required  system  performance  while  making  economical  demands  upon  personnel  resources,  minimizing 
life  cycle  costs,  and  managing  the  risk  of  loss  or  injury  to  personnel,  equipment,  or  the  environment. 

Identifying  optimal  HSI  solutions  within  the  life  cycle  management  process  is  a  complex  endeavor 
requiring  a  holistic  understanding  of  the  relationships  between  the  nine  distinct  conceptual  HSI 
"domains"  (i.e.,  Manpower,  Personnel,  Training,  Human  Factors  Engineering,  Survivability,  Habitability, 
Safety,  Environment,  and  Occupational  Health)  and  the  ability  to  manage  interactions  between  these 
domains. 

The  711th  Human  Performance  Wing  Human  Systems  Integration  Directorate  (711  HPW/HP)  is  working 
to  address  a  perceived  capability  gap  within  AF  program  management  and  acquisitions  arenas,  by:  1) 
emphasizing  the  importance  of  including  HSI  considerations  in  the  system  design  and  development 
process;  and  2)  enhancing  the  understanding  of  HSI  tradeoffs. 

At  the  direction  of  711  HPW/HP,  the  Survivability/  Vulnerability  Information  Analysis  Center  (SURVIAC) 
developed  the  HSI  Tradeoff  Demonstration  Model  to  build  awareness  of  HSI  and  HSI-related  engineering 
concepts.  The  model,  built  in  the  form  of  a  software  tutorial,  also  develops  the  ability  of  program 
management  professionals  to  more  easily  identify  potential  HSI  tradeoffs  in  program  management  and 
systems  engineering  functional  areas,  and  demonstrates  how  these  HSI-related  tradeoffs  influence 
system  level  tradeoffs. 

Section  2.0  of  this  report  provides  a  top-level  overview  of  the  model.  Section  3.0  contains  a  step-by- 
step  explanation  of  the  mathematical  basis  for  two  HSI  tradeoff  exercises  contained  in  the  model. 
Summaries  of  two  usability  tests  performed  on  the  model  are  presented  in  Section  4.0. 

2.0  Overview  of  the  HSI  Tradeoff  Demonstration  Model 

The  HSI  Tradeoff  Demonstration  Model  (hereinafter  designated  as  "the  model")  is  an  HTML  application 
that  enables  program  managers  and  engineers  to  familiarize  themselves  with  HSI  in  an  interactive 
fashion,  first  by  reading  a  top-level  introduction  to  HSI  and  the  constituent  domains,  and  then  by 
stepping  through  a  series  of  HSI-related  tradeoffs  on  two  notional  development  programs. 
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Upon  opening  the  application,1  the  user  is  presented  with  the  home  screen  shown  in  Figure  1.  The 
screen  has  three  "clickable"  buttons: 

•  WhatisHSI? 

•  TSA  Scenario;  and 

•  UAS  Scenario 


Figure  1  Home  Screen 


2.1  What  is  HSI? 

The  'What  is  HSI?'  section  gives  an  introduction  to  HSI  and  its  constituent  domains  and  helps  the  user 
understand  that  there  are  tradeoffs  between  each  domain.  Background  information,  definitions,  and 
examples  can  be  found  here. 

2.2  Transportation  Security  Adminstration  (TSA)  Scenario 

This  section  is  a  mock  scenario  to  improve  airport  passenger  screening  through  the  purchase  of  a 
surveillance  system.  Two  systems  are  available:  "Touchless  Invader"  and  "Spinex."  Users  must  also 
select  appropriate  amounts  of  manpower,  personnel  aptitude,  and  training.  The  goal  is  to  find  a  solution 
that:  (1)  remains  within  budget;  and  (2)  meets  minimum  requirements  for  passenger  throughput  and 
detection  rate.  Users  are  presented  with  the  corresponding  results  of  their  choices  and  informed 
whether  or  not  they  have  passed/failed  or  found  the  optimal  solution. 

2.3  Unmanned  Aircraft  System  (UAS)  Scenario 

This  section  is  based  on  the  use  of  UAS  by  local  law  enforcement.  Users  must  choose  a  configuration  for 
control  station  monitors  that  will  optimize  UAS  operator  workload  and  remain  within  a  budget.  As  in  the 
TSA  scenario,  they  are  then  presented  with  the  corresponding  results  of  their  choices,  and  informed 
whether  or  not  they  have  passed/failed  or  found  the  optimal  solution. 


1  Requirements  for  installation  of  the  model  and  opening  instructions  are  found  in  Appendix  C. 
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2.4  Navigation 

To  navigate  within  the  application,  the  user  clicks  on  arrows  at  the  bottom  of  the  screen  to  go  to  the 
next  or  previous  screen.  To  go  to  the  home  screen  and  start  over,  he/she  clicks  the  'Home'  button  in 
either  the  bottom  left  or  top  left  corners.  There  is  also  a  "breadcrumb  trail"  at  the  top  that  will  indicate 
where  the  user  currently  is  within  the  application.  Clicking  on  any  previous  section  will  take  the  user 
back  to  that  section  of  the  scenario  (See  Figure  2). 


Figure  2  Navigation 

2.5  User  Input 

Throughout  the  application,  the  user  is  prompted  to  provide  input.  Checkboxes  allow  for  multiple 
selections  using  the  mouse  (Figure  3).  Radio  buttons,  on  the  other  hand,  only  allow  the  user  one 
selection,  also  selected  using  the  mouse  (Figure  4).  Finally,  the  drag  and  drop  interface  enables  the  user 
to  select  a  component  by  left-clicking  and  holding,  dragging  the  mouse  to  the  desired  destination,  and 
then  releasing  the  button  (Figure  5). 
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Figure  3  Check  Boxes 
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Home  y  TSA  Scenario  Introduction  y  Systems  Analysis  Training  Selection 


TSA  Scenario:  System  Selection 


Considering  the  choices  you  made,  it  is  now  time  to  pick  which  system  you 
would  like  to  install  at  airports.  Recall  that  you  are  trying  to  maximize 
detection  reliability,  minimize  manpower,  and  minimize  your  overall 
cost. 


Personnel  Choice 

Score  80 


Training  Choice 


Choose  the  system 
operating  cost. 


Left  Click  to  for  Single 
Selection 


60  hours  of  training 
he  total  system  performance  and 


System 

Cost 

Manpower 

Throughput 

®  Tou'-’.iess  Invader 

$3,750,000  ea. 

2  minimum 

60  max  per 

•  Spinex 

$3,500,000  ea. 

3  minimum 

50  max  per 

System  Cost:  $18,750,000 


£ 


Figure  4  Radio  Buttons 


Home  UAS  Scenario  >  Introduction  Analysis  >  Monitors  Design 


UAS  Scenario:  Design 

Please  design  your  display  by 
placing  the  appropriate  screens 
on  the  monitors.  Optimize  the 
placement  of  each  screen  by 
considering  what  the  operator 
will  use  the  most  or  need  acces; 
to  in  a  challenging  situation. 

Map  and  PFD  are  full  screen. 
Comms,  EP,  Health  Status,  and 
Chat  are  half  screens. 

Left-Click  and  Hold  to  Drag 
and  Drop  Objects 


_ X 

Drop  here 

1 

2 

3 

PFD  | 

Drag  the  componSntejo  the  monitors  above  to  place  them. 


Map 

PFD 

■ 

Comms 

EP 

Health  Status 

Chat 

1 

■ 

Figure  5  Drag  and  Drop 


2.6  Optimal  Solutions 

Each  scenario  has  an  optimal  solution  that  the  user  attempts  to  achieve.  These  solutions  are  driven  by 
different  factors  depending  on  the  scenario. 

2.6.1  TSA  Scenario  Solution 

In  this  scenario,  there  are  three  conditions  that  must  be  met  in  order  for  the  user  to  be  successful.  First, 
he/she  must  procure  and  operate  a  system  within  a  budget  of  $20  million.  This  means  choosing 
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personnel  skill  levels,  training,  HSI  analyses,  and  a  system  that  all  fall  within  this  budget.  Second,  a 
throughput  of  at  least  250  passengers  per  hour  must  be  achieved.  Finally,  the  system  must  have  at  least 
a  97%  detection  rate  to  be  a  viable  solution. 

An  optimal  solution  is  one  that  combines  lowest  cost  with  most  efficient  operation  and  training  time.  In 
this  scenario,  the  optimal  solution  is  as  follows  (does  not  take  into  account  purchase  of  available  HSI 
analyses): 

•  Aptitude:  Score  of  60 

•  Training:  60  hours 

•  System:  Touchless  Invader 

2.6.2  UAS  Scenario  Solution 

The  UAS  scenario  has  two  conditions  that  must  be  met  in  order  to  have  a  successful  system.  The  first  is 
to  procure  a  system  within  a  budget  of  $3  million.  The  second  condition  is  to  minimize  the  operator's 
visual  workload.  Once  these  conditions  are  met,  the  user  will  have  a  successful  solution. 

An  optimal  solution  stays  within  budget  and  has  the  lowest  visual  workload.  In  this  scenario,  there  is  a 
best  solution  for  each  screen  configuration.  Figure  6  shows  the  best  solution  for  each  configuration. 
Configuration  C  is  the  overall  optimal  solution  for  all  screen  configurations. 


Comms 

PFD 

Map/Nav 

Health  Status 

Monitor  Configuration  A 


Health  Status/ 
Comms 

PFD 

Map 

Monitor  Configuration  B 


Health  Status 


Map/Nav 


Monitor  Configuration  C  (Overall  Optimal) 


Figure  6  Optimizing  the  Monitor  Configuration 


3.0  Scenarios  and  Tradeoff  Model  Details 

This  section  describes  the  two  scenarios  in  detail,  and  provides  the  mathematical  models  that  are  used 
to  score  them. 

3.1  TSA  Scenario 

The  aim  of  the  TSA  scenario  is  to  improve  airport  passenger  screening  through  the  purchase  of  a 
surveillance  system.  The  user  may  perform  or  purchase  HSI  analysis  to  choose  a  system  that  meets 
performance  goals.  The  three  goals  identified  for  this  scenario  are: 

•  Stay  within  a  budget  of  $20M 

•  Maintain  throughput  of  250  passengers/hour 

•  Meet  required  reliability  of  97% 
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Major  HSI  domains  and  activities  assessed  in  this  scenario  include  choices  of:  (1)  initial  personnel  skill 
level ;  (2)  training;  and  (3)  amount  of  manpower  employed.  Manpower  requirements  are  determined  by 
the  user's  choice  of  surveillance  system  (i.e.,  Touchless  Invader  or  Spinex).  Throughput  capability  varies 
as  a  function  of  personnel  skill  levels,  which  are  derived  from  incoming  skill  level  ( aptitude  scores )  and 
additional  training  required.  Aptitude  scores  and  additional  training  are  variables  chosen  by  the  user 
when  establishing  the  parameters  of  the  scenario. 

This  scenario  shows  the  importance  of  assessing  HSI  domains  in  conjunction  with  other  risks  and 
decision-making  activities.  HSI  is  intrinsically  integrated  with  cost  and  performance.  To  design  a  system 
correctly,  additional  studies,  analyses,  and  design  considerations  may  be  necessary. 

Since  this  is  a  hypothetical  trade-off  analysis,  SURVIAC  first  created  a  dataset  for  total  system  cost  (TCS): 

TCs  —  nCs  +  {Cj  +  Sp)No 

where  n  is  the  number  of  systems,  Cs  is  the  cost  of  the  system,  CT  is  the  cost  of  training  each  operator,  SP 
is  the  burdened  salary  for  each  operator  (for  the  specified  period)  and  N0  is  the  number  of  operators 
required. 

System  cost  (Cs)  is  assumed  to  be  a  given  value  based  on  the  retail  cost  established  by  the  manufacturer 
(based  on  scenario)  and  the  manpower  cost  to  operate  the  system.  Training  Cost  (CT)  is  a  linear  function 
of  cost  per  hour,  with  x  representing  number  of  hours  and  Ch  representing  minimum  cost  per  hour 
($200). 
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Figure  7  Training  Cost  as  a  Function  of  Training  Hours 


Personnel  Salary  (SP)  is  a  function  of  the  individual's  initial  skill  level  as  reflected  by  their  aptitude  score 
(x0).  Salary  increases  in  direct  proportion  to  higher  aptitude  scores. 

SP  =  2000xo  -  50,000 
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Figure  8  Personnel  Salary  versus  Aptitude  Score 

The  user  must  then  make  a  system  choice  between  two  systems  (shown  below  in  Table  1). 

Table  1 TSA  System  Choices 


System 

Cost  ($M) 

Minimum 

Manpower 

Throughput 
(People  per  hr.) 

Touchless  Invader 

3.5 

2 

60 

Spinex 

3.0 

3 

50 

SURVIAC  integrated  a  Training  Needs  Analysis  and  a  Throughput  and  Manpower  Analysis  into  the  TSA 
scenario  to  assist  with  decision  making,  if  the  user  chooses.  Because  of  the  importance  in  presenting 
tradeoffs,  these  analyses  are  only  made  available  for  an  additional  cost.  These  analyses  emphasize  the 
message  that  additional  upfront  investment,  through  studies,  analysis,  or  other  design  considerations,  is 
often  needed  to  design  a  system  correctly. 

The  Training  Needs  Analysis  provides  the  user  with  information  on  the  inverse  relationship  between 
initial  personnel  skill  levels  and  the  amount  of  subsequent  training  needed  to  meet  the  required  97% 
detection  rate  (See  Figure  9). 
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Figure  9  Training  Hours  versus  Aptitude  Score  for  97%  Detection  Level 
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Throughput  and  Manpower  Analysis  provides  analysis  on  the  number  of  people  who  can  be  screened 
per  hour  while  still  meeting  the  required  detection  rate.  Combining  this  information  with  the  Training 
Needs  Analysis  provides  the  user  with  a  required  number  of  systems  and  manning  levels  to  meet  the 
throughput  goal. 


System  A  (Touchless  Invader) 

System  B  (Spinex) 

Training 

Hrs 

Personnel 

Aptitude 

Throughput 

Reliability  (%) 

Initial  Cost  (lyr) 

Training  Hrs 

Personnel 

Aptitude 

Throughput 

Reliability 

(%) 

Initial  Cost  (lyr) 

16 

100 

60 

97 

$  21,532,000 

16 

100 

50 

98 

$19,798,000 

40 

100 

60 

97 

$  21,580,000 

40 

100 

50 

98 

$19,870,000 

40 

80 

60 

97 

$  21,180,000 

40 

80 

50 

98 

$19,270,000 

60 

100 

60 

97 

$  21,620,000 

60 

100 

50 

98 

$19,930,000 

60 

80 

60 

97 

$  21,220,000 

60 

80 

50 

98 

$19,330,000 

60 

60 

60 

97 

$  20,820,000 

60 

60 

50 

98 

$18,730,000 

Figure  10  Acceptable  Solutions  and  Costs 


Appendix  D  contains  all  data  and  information  used  in  the  creation  of  this  model.  This  information  is 
notional  and  only  valid  within  the  context  of  this  scenario  and  model. 

3.2  UAS  Scenario 

The  UAS  scenario  demonstrates  the  importance  of  HSI  through  the  design  of  a  display  system  for 
operating  unmanned  aircraft  in  a  law  enforcement  capacity,  such  as  border  patrol,  suspect  tracking,  or 
location  monitoring.  In  this  scenario,  there  is  an  increased  potential  for  system  loss  due  to  operator 
error. 

This  scenario  focuses  on  three  objectives: 

•  Procure  a  system  within  a  budget  of  $3M 

•  Improve  the  design  of  the  system 

•  Minimize  operator's  visual  workload 

The  user  must  design  a  new  operator  workstation,  choosing  the  physical  configuration  of  the  display 
monitors,  location  of  output  data  displayed,  operator  skill  level,  and  amount  of  training  to  be  provided. 
Impact  on  operator  workload  is  determined  by  these  choices. 

Users  have  the  opportunity  to  purchase  up  to  two  types  of  HSI  analyses  for  this  scenario:  Task  and 
Workload  Analysis  and  Training  Needs  Analysis.  Task  and  Workload  Analysis  defines  critical  information 
requirements  and  determines  how  visual  workload  is  calculated  through  screen  placement.  Training 
Needs  Analysis  measures  the  workload  associated  with  each  potential  operator's  attributes  and  skill 
level. 

Users  have  the  option  to  choose  up  to  six  control  station  display  screens.  A  description  of  each  screen  is 
shown  in  Figure  11. 
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Health 
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Chat 


Emergency  Procedures  (EPs)  are  the  activities  or  checklists  that  the  operator 
follows  when  an  emergency  situation  occurs  (e.g.  engine  failure). 

Primary  Flight  Display  (PFD)  provides  basic  flight  Information  In  one  location. 


Comms  are  the  communication  links  allowing  the  control  station  to  communicate 
through  the  aircraft  with  other  personnel. 

Health  Status  presents  Information  on  the  state  of  different  aircraft  components 
(e.g.  oil  level). 

Map  or  navigation  screen  is  the  visual  representation  of  the  area  where  the 
aircraft  Is  transferring  to  or  flying  over. 

Chat  Is  the  communication  method  that  the  operator  uses  to  textually 
communicate  with  other  Interested  or  Involved  parties. 


Figure  11  Display  Screen  Options 


The  user  is  then  provided  with  the  HSI-related  analyses  to  provide  information  on  the  required  display 
system  capabilities.  As  part  of  the  Task  and  Workload  Analysis,  the  user  discovers  that  the  HSI  team  will 
conduct  a  cognitive  task  analysis  to  identify  critical  information  requirements  for  UAS  displays.  Based  on 
this  cognitive  task  analysis,  only  four  of  the  six  screens  are  identified  as  visual  workload  drivers:  primary 
flight  display,  moving  map,  communications,  and  emergency  procedures.  These  four  screens  are 
included  in  the  estimation  of  visual  workload. 


To  facilitate  matrix  representation  in  the  mathematical  model,  each  of  these  mission  critical  tasks  is 
given  an  index,  as  shown  in  Table  2: 


Table  2  Task  Index  Assignments 


Task 

Index 

Primary  Flight  Display 

0 

Moving  Map 

1 

Communications 

2 

Emergency  Procedures 

3 

The  probability  that  the  operator  will  transition  between  tasks  (normally  on  separate  screens)  is  a  major 
driver  in  the  visual  workload.  SURVIAC  estimated  the  probability  of  transitioning  between  tasks  as 
follows  (Table  3): 

Table  3  Transition  Probability 


Task 

0  12  3 

0 

0.0000 

0.6000 

0.3000 

0.1000 

1 

0.5000 

0.0000 

0.4000 

0.1000 

2 

0.7000 

0.2000 

0.0000 

0.1000 

3 

0.2000 

0.2000 

0.6000 

0.0000 

These  probability  values  are  notional  for  the  purposes  of  the  demonstration  scenario  and  are  assumed 
valid  only  in  the  context  of  the  model. 

Users  are  informed  that  increasing  the  number  of  display  monitors  also  increases  the  visual  workload  for 
each  operator.  Available  configurations  for  display  monitors  are  shown  in  Figure  12. 
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Two  Monitor 


Three  Monitor 


Monitor  1  Monitor  2 


Four  Monitor  (A) 


Monitor  1 

Monitor  2 

Monitor  3 

Monitor  4 

Monitor  1 

Monitor  2 

Monitor  3 

Four  Monitor  (B) 

Mnnitnr  7 

Monitor  1 

Monitor  3 

Monitor  4 

Figure  12  Screen  Configuration 

Measurement  information  for  head/eye  movement  between  displays  and  screens  is  presented. 
Distances  between  tasks  are  shown  in  Figure  13: 


Two  Monitor  Three  Monitor 


Four  Monitor  (A) 


Four  Monitor  (B) 


Figure  13  Distances  between  Tasks 

(Note:  Communications  and  health  status  tasks  are  combined  on  a  single  monitor) 

In  addition  to  the  physical  configuration,  the  layout  of  the  tasks  also  impacts  the  distance  between  the 
tasks.  A  matrix  of  distances  between  tasks  was  calculated  for  every  screen  configuration  and  task  layout 
combination.  The  two  monitor  configuration  resulted  in  six  distance  matrices,  the  three  monitor 
configuration  resulted  in  12  distance  matrices,  and  the  four  monitor  (both  A  and  B)  configurations 
resulted  in  72  distance  matrices.  Screen  layouts  and  corresponding  distance  matrices  are  located  in 
Appendix  D. 


Time  to  transition  between  tasks  is  a  function  of  distance  between  the  tasks  and  is  estimated  in 
Equation  1,  where  £[T/fc]  is  the  mean  expected  transition  time  between  task  /  and  task  j  for  screen 
layout  k,  dijk  is  the  distance  between  task  /  and  task  j  for  screen  layout  k,  and  m  and  b  are  constants. 
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Equation  1 


E[Tijk]  ~  m(^ijk  +  & 

The  screen  layout  index  k  denotes  the  physical  configuration  of  the  monitors  (2,  3,  or  4),  and  the  layout 
of  the  tasks  on  the  screens.  Constants  m  and  b  are  identical  between  monitor  configuration  and  task 
layout,  with  m  set  to  a  value  of  1  and  b  to  a  value  of  0  for  demonstration  purposes.  The  constants  can 
be  adjusted  to  model  more  realistic  transition  times  based  on  regression  of  human  head/eye 
movement.  The  mathematical  model  is  structured  so  that  placing  tasks  with  a  high  probability  of 
transition  on  screens  further  from  each  other  results  in  a  higher  visual  workload. 

The  Training  Needs  Analysis  provides  users  with  three  potential  operator  pools,  each  having  a  different 
initial  skill  level  (aptitude).  Operators  must  obtain  additional  training  to  effectively  operate  the  UAS. 
Three  discrete  values  are  available  for  both  training  time  and  operator  aptitude.  Training  time  varies 
between  50,  100,  and  150  hours.  Operators  with  a  skill  level  similar  to  that  of  an  experienced  gamer  are 
assigned  aptitude  values  of  45,  sport  pilot  skill  levels  are  assigned  an  aptitude  value  of  60,  and 
commercial  pilot  skill  levels  are  assigned  an  aptitude  value  of  75.  Completion  time  for  each  task  is  a 
function  of  aptitude  and  training  time  and  can  be  estimated  using  Equation  2  (based  on  the  power  law 
of  practice): 

E[Ti\  —  aiN~bi  +X  Equation  2 

where  at  and  bt  are  constants  for  task  /,  N  is  training  time,  and  X  is  operator  aptitude.  The  values  were 
selected  to  yield  a  wide  range  of  possible  task  completion  times.  Constants  at  and  bt  are  selected  so 
that  the  completion  time  for  each  task  follows  the  expected  trend  that  higher  training  times  and 
aptitude  values  result  in  lower  task  completion  times.  For  the  purposes  of  the  demonstration  model,  the 
constants  are  assumed  to  be  identical  between  tasks.  Constants  can  be  refined  to  more  realistically 
model  the  impact  of  training  and  aptitude  on  operator  performance.  A  plot  of  task  completion  times 
using  the  notional  training  times,  aptitudes,  and  constants  can  be  seen  in  Figure  14: 


Experienced  Gamer  Sport  Pilot  Commercial  Pilot 


50  60  70  80  90  100  110  120  130  140  150 

Operator  Training  [hr] 

Figure  14  Expected  Task  Completion  Time 

The  task  completion  times  are  clearly  offset  to  show  the  impact  of  aptitude.  A  limiting  probability  is 
calculated  to  determine  the  long-term  probability  that  the  operator  will  be  in  a  given  task  when  starting 
to  transition  to  another  task  (Table  4). 
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Table  4  Limiting  Probability 


Task 

Limiting  Probability 

Primary  Flight  Display 

0.3516 

Moving  Map 

0.2839 

Communications 

0.2736 

Emergency  Procedures 

0.0909 

Since  the  limiting  probabilities  are  calculated  from  the  transition  matrix  in  Table  3,  these  values  were 
also  notional  and  assumed  valid  only  in  the  context  of  the  model.  The  mathematical  model  is  structured 
so  that  tasks  with  a  high  limiting  probability  and  a  high  completion  time  result  in  a  higher  workload. 


Upon  completion  of  the  HSI-related  analyses,  the  user  designs  the  UAS  display  system,  choosing  the 
number  of  monitors,  layout,  type  of  operator  (aptitude),  and  amount  of  training.  The  workload  is 
calculated  for  each  combination  of  monitor  configuration,  screen  layout,  operator  training,  and  operator 
aptitude  using  Equation  3: 


E[W]  =  Ilo{*iWiKlloPijE[Tvk])) 


Equation  3 


where  E\W]  is  the  expected  workload,  7r;  is  the  limiting  probability  for  task  /  (Table  3),  E[Tt ]  is  the 
expected  completion  time  of  task  /  (Eq  2),  Py  is  probability  of  transitioning  from  task  /  to  task  j  (Table  2), 
and  £’[7’iyk]  is  the  expected  transition  time  between  task  /  and  task  j  for  screen  layout  k  (Eq  1).  An 
example  of  the  estimated  workload  calculations  can  be  seen  in  Figure  15: 


Layout  A 

-Experienced  Gamer - Sport  Pilot - Commercial  Pilot 


50  60  70  80  90  100  110  120  130  140  150 

Training  Time  [hr] 


Layout  C 

-Experienced  Gamer - Sport  Pilot - Commercial  Pilot 


Training  Time  [hr] 


Figure  15  Example  Workload  Output 


Figure  15  shows  a  comparison  of  the  estimated  workload  as  a  function  of  screen  layout,  aptitude,  and 
training.  Due  to  the  closer  proximity  of  the  highest  probability  tasks  (primary  flight  display  and  moving 
map),  the  estimated  workload  for  Layout  C  is  lower  than  Layout  A.  Additionally,  the  workload 
dramatically  reduces  as  longer  training  times  are  used  for  operators  with  lower  initial  aptitudes.  At  a 
training  time  of  150  hours,  the  visual  workloads  for  each  type  of  operator  approach  a  single  value.  The 
plots  also  illustrate  the  diminishing  returns  of  longer  training  times  for  operators  with  a  high  initial 
aptitude  (commercial  pilot).  The  mathematical  model  captures  the  desired  visual  workload  behavior, 
based  on  the  calculations  for  the  complete  set  of  configuration,  layout,  training,  and  aptitude  input 
variables.  To  succeed  or  "pass"  this  scenario,  the  user  must  have  spent  less  than  $3M  and  have  an 
acceptable  level  of  workload.  Within  the  context  of  the  scenario,  an  acceptable  average  workload  level 
is  defined  as  70-80%.  The  user  is  rewarded  with  an  animated  graphic  depicting  his/her  success  or  failure 
to  meet  the  requirements. 
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4.0  User  Testing 

SURVIAC  performed  two  user  tests  of  the  HSI  Tradeoff  Demonstration  Model.  The  first  test,  conducted 
February/March  2012,  focused  on  an  early  prototype  of  the  model.  Six  users  from  SURVIAC  and  711 
HPW/HP  participated  in  this  test.  The  results  were  used  to  make  improvements  to  the  final  version  of 
the  model,  which  was  evaluated  by  six  users  in  June  2012  (five  of  these  had  participated  in  the  first  test.) 
Each  test  session  consisted  of  pre-  and  post-testing  of  users'  HSI  domain  knowledge,  a  model  walk¬ 
through,  and  the  completion  of  a  user  survey  assessing  the  model's  usability. 

4.1  Training  Effectiveness 

At  the  beginning  of  each  test  session,  SURVIAC  administered  a  pre-test  (Appendix  E)  to  identify  users' 
starting  knowledge  about  the  HSI  domains.  At  the  end  of  each  session,  a  post-test,  consisting  of  the 
same  questions  was  given.  By  comparing  results,  it  was  possible  to  determine  if  the  model  provided 
information  transfer  to  the  users.  The  multiple  choice  pre-  and  post-tests  focused  on  the  ability  of  the 
participants  to: 

•  Understand  and  identify  the  nine  different  Air  Force  HSI  domains 

•  Understand  the  definition  of  HSI 

•  Understand  the  importance  of  HSI  analyses 

•  Understand  the  criticality  and  difficulty  of  performing  HSI-related  tradeoffs 

In  general,  the  users  who  participated  in  the  study  exhibited  a  relatively  good  understanding  of  HSI 
domains  on  the  pre-test.  That  fact,  along  with  the  small  number  of  participants  precludes  any  significant 
conclusions  about  the  model's  training  effectiveness.  However,  in  both  sessions,  the  post-test  results 
showed  an  improvement  in  the  number  of  correct  answers. 

4.2  Model  Walk-Through  and  Usability  Assessment 

After  completing  the  HSI  knowledge  pre-test,  participants  were  provided  with  an  electronic 
questionnaire  (Appendix  G)  to  collect  background  information  and  to  guide  their  evaluation  of  the 
model.  A  model  walk-through  was  performed  by  each  user  while  the  SURVIAC  model  development  team 
observed.  The  users  evaluated  the  functionality  and  usability  of  the  model  in  understanding  the  goal  of 
HSI  and  the  importance  of  performing  HSI-related  tradeoffs.  A  five-point  Likert  scale  was  used  to  identify 
users'  satisfaction  or  agreement  with  each  usability  assessment  question;  one  represented  the  low  or 
negative  end  of  the  scale  and  five  represented  the  high  or  positive  end  of  the  scale.  In  addition  to  the 
rating  scale,  the  SURVIAC  team  requested  the  users  provide  comments  or  additional  feedback  to  explain 
the  numerical  rating.  An  example  of  the  rating  scale  is  provided  in  Figure  16. 


Ratings  Comments 

2  Please  click  on  the  What  is  HSI?"  label  on  the  Home  page 

2.1.  The  information  is  presented  in 

Choose  an  item  ▼ 

a  clear  and  concise  manner  If 

not,  how  would  you  recommend 

1  -  Strongly  Disagree 

we  present  background/basic 

2  -  Disagree 

HSI  information? 

3  -  Neutral 

4 -Agree 

5 -Strongly  Agree 

Figure  16  Usability  Test  1  Rating  Scale 

During  the  first  user  test,  the  SURVIAC  team  identified  additional  opportunities  to  improve  the 
background  HSI  information  and  the  presentation  of  the  TSA  and  UAS  scenarios.  The  second  test  was 
used  to  validate  and  confirm  usability  and  content  of  the  model.  Feedback  from  the  second  test  focused 
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on  grammatical  and  software  errors  and  the  recommendation  to  add  a  screen  (Figure  17)  discussing  the 
importance  and  difficulty  of  conducting  tradeoffs. 


Figure  17  Tradeoff  Complexity  Screen 

The  critical  feedback  from  this  usability  testing  indicated  that  the  tool  and  scenarios  were  useful  in 
identifying  HSI  related  tradeoffs  for  different  types  of  systems.  The  test  participants  found  that  the 
analysis  provided  additional  information  to  assist  with  decision  making. 
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5.0  Summary 

The  HSI  Tradeoff  Model  was  developed  by  SURVIAC  as  a  tool  to  educate  program  managers  and  systems 
engineering  professionals  about  Human  Systems  Integration  and  its  importance  within  the  Air  Force's 
life  cycle  management  processes. 

In  addition  to  providing  a  greater  understanding  of  HSI  concepts,  the  model  demonstrates  the 
importance  of  HSI  tradeoffs  and  how  the  nine  HSI  domains  interface  with  each  other  and  the  systems 
engineering  functions  of  cost,  schedule,  and  performance. 

The  HSI  Tradeoff  Model  consists  of  one  tutorial  and  two  scenario-based  concept  application  modules 
that  allow  the  user  to  apply  basic  HSI  analysis  information,  explore  options,  and  make  decisions  about 
the  design  and  acquisition  of  a  new  system  that  will  potentially  enhance  the  personnel  and  system 
performance  and  survivability.  It  provides  a  graphical,  interactive  representation  of  the  tradeoffs 
involved  in  incorporating  HSI  concepts  into  the  life  cycle  management  process. 

By  providing  this  model  to  program  managers  and  systems  engineering  professionals,  the  711  HPW/HP 
further  advances  an  understanding  of  concepts  essential  to  Department  of  Defense  and  Air  Force  life 
cycle  management  processes.  These  concepts,  when  applied  effectively,  will  have  both  an  immediate 
and  long-term  effect  on  personnel  and  system  performance  and  survivability. 
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Appendix  A:  Acronyms 


711  HPW/HP  711th  Human  Performance  Wing/Human  Systems  Integration  Directorate 
AF  Air  Force 

DoDI  Department  of  Defense  Instruction 

EP  Emergency  Procedure 

HSI  Human  Systems  Integration 

PFD  Primary  Flight  Display 

SURVIAC  Survivability  Vulnerability  Information  Analysis  Center 

TSA  Transportation  Security  Administration 

UAS  Unmanned  Aircraft  System 

USAF  United  States  Air  Force 
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Appendix  C:  Model  Installation  and  Operation  Requirements 

The  HSI  Tradeoff  Demonstration  model  requires  the  following  for  optimal  use: 

•  Operating  System:  Windows  XP  SP2,  Windows  Vista,  Windows  7,  Mac  OS  X  10.4.8  or  above. 

•  Processor:  Windows:  Intel  Pentium  III  or  higher,  Mac:  Intel  Core  Duo. 

•  Memory:  128  MB  or  greater 

•  Software:  Silverlight  4.0  or  higher,  installation  can  be  found  at 
http://www.microsoft.com/getsilverlight/Get-Started/lnstall/Default.aspx 

•  Screen  Resolution:  1280  x  768  or  higher  otherwise  you  might  notice  some  parts  of  the  application 
being  cut  off.  If  this  is  the  case,  follow  the  directions  on  the  first  page  of  the  application  to  resolve 
the  issue. 

To  open  the  stand-alone  version  of  the  711th  Tradeoff  Demo  model,  double  click  on  the  '711th.html'  file 

(Figure  18).  A  browser  window  should  open  with  the  application. 


Figure  18  Opening  the  Application 
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Appendix  D:  Scenario  Equations  and  Calculations 


The  Design  Scenario  calculations  and  TSA  Scenario  calculations  spreadsheets  are  available  upon  request. 
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Appendix  E:  Usability  Test  Artifacts  -  HSI  Tradeoff  Tool  TSA  Scenario  Pre-Test 


HSI  Marketing  Demonstration  Tradeoff  Tool:  PRE-TEST 

Please  complete  the  questions  below. 


Name: 


Date: 


6/25/2012 


1.1.  Match  the  following  terms  with  the 

appropriate  definition. 

A.  Training 

B.  Personnel 

C.  Human  Factors  Engineering 

D.  Manpower 

E.  Safety 

F.  Occupational  Health 

Choose  an 
item. 

Design  for/around  the  human 

Choose  an 
item. 

Prevent  long  term  injuries  and 
disabilities 

Choose  an 
item. 

Number  of  spaces  or  billets 

Choose  an 
item. 

Skill  enhancement 

Choose  an 
item. 

Mishap  and  injury  prevention 

Choose  an 
item. 

Types  of  faces 

1.2.  HSI  is... 

A.  Human  Systems  Interaction 

B.  Human  Factors  Engineering 

C.  Human  Systems  Integration 

D.  Human  Systems  Interdependence 

Choose  an  item. 

1.3.  HSI  in  the  Air  Force  has  domains 

1.  Choose  an  item. 

1.4.  HSI  identifies  interdependencies  and 

tradeoffs  among... 

2. 

3.  Choose  an  item. 

1 .5.  HSI  helps  to  ensure  that  the  human  is 

considered  on  par  with  the  hardware  and 
software  during  system  design  and 
development. 

4.  Choose  an  item. 

1.6.  Performing  HSI  activities  supports  increased 

human  performance 

5.  Choose  an  item. 

1 .7.  Effective  consideration  of  HSI  helps  to 

reduce  total  ownership  costs  and  safety 
mishaps. 

6. 

7.  Choose  an  item. 
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Appendix  F:  Usability  Test  Artifacts  -  HSI  Tradeoff  Tool  TSA  Scenario  Post-Test 

HSI  Marketing  Demonstration  Tradeoff  Tool:  POST-TEST 

Please  complete  the  questions  below. 


Name: 


Date: 


6/25/2012 


1 .8.  Match  the  following  terms  with  the 

appropriate  definition. 

G.  Training 

H.  Personnel 

I.  Human  Factors  Engineering 

J.  Manpower 

K.  Safety 

L.  Occupational  Health 

Choose  an 
item. 

Design  for/around  the  human 

Choose  an 
item. 

Prevent  long  term  injuries  and 
disabilities 

Choose  an 
item. 

Number  of  spaces  or  billets 

Choose  an 
item. 

Skill  enhancement 

Choose  an 
item. 

Mishap  and  injury  prevention 

Choose  an 
item. 

Types  of  faces 

1.9.  HSI  is... 

E.  Human  Systems  Interaction 

F.  Human  Factors  Engineering 

G.  Human  Systems  Integration 

H.  Human  Systems  Interdependence 

Choose  an  item. 

1.10.  HSI  in  the  Air  Force  has  domains 

8.  Choose  an  item. 

1.11.  HSI  identifies  interdependencies  and 
tradeoffs  among... 

9. 

10.  Choose  an  item. 

1.12.  HSI  helps  to  ensure  that  the  human  is 

considered  on  par  with  the  hardware  and 
software  during  system  design  and 
development. 

11.  Choose  an  item. 

1.13.  Performing  HSI  activities  supports  increased 
human  performance 

12.  Choose  an  item. 

1.14.  Effective  consideration  of  HSI  helps  to 
reduce  total  ownership  costs  and  safety 
mishaps. 

13. 

14.  Choose  an  item. 
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Appendix  G:  Usability  Test  Artifacts  -  HSI  Tradeoff  Tool  User  Test 

Human  Systems  Integration  Marketing  Demonstration  Tradeoff  Tool 

Section  1:  Background  Information 

The  information  you  provide  in  this  section  is  for  statistical  purposes  only.  We  ask  that  you  provide  your 
name  and  answer  the  following  background/demographic  questions. 


Name: 

Comments 

1.  Background/Demographics 

1.15.  Do  you  work  in  the  field  of  HSI? 

If  yes,  please  describe 
activities. 

2.  Choose 

an  item. 

3.  1.2  Are  you  familiar  with  the 

domains  of  Air  Force  HSI? 

4. 

an  item. 

5.  1.3  Are  you  government  or 

contractor? 

6.  Choose 

an  item. 

7.  1.4  Have  you  participated 

in  developing  marketing  tools  or 
materials  in  the  past?  If  yes,  for 
what  industry. 

8.  Choose 

an  item. 

Section  2:  Time 

Please  record  your  start  and  end  times.  Start  with  the  “What  is  HSI?”  link  and  continue  to  the  TSA  Scenario. 
Demonstration  Link:  http://www.bahdayton.com/71 1th 


Time  Initiated:  From  scenario  initiation 

9. 

Time  Completed:  Results  screen 

10. 

Section  3:  Usability/Tasks  Walkthrough 

Please  read  the  instructions  below  and  provide  your  feedback  on  the  tasks  as  they  are  completed.  For  this 
section  of  the  questionnaire,  you  will  go  through  an  example  scenario  of  identifying  HSI-related  tradeoffs. 

Please  rate  the  applicable  statements  on  a  5-point  scale  with  1  representing  the  low  or  negative  end  of 
the  scale  and  5  as  the  high  or  positive  end  of  the  scale  and  provide  comments  to  support  your  rating. 


Ratings 

Comments 

2.  Please  click  on  the  “What  is  HSI?”  label  on  the  Home  page. 

2.1.  The  information  is  presented  in 

a  clear  and  concise  manner.  If 
not,  how  would  you  recommend 
we  present  background/basic 

HSI  information? 

11. 

an  item. 

2.2.  The  definition  of  HSI  accurately 

represents  the  functions 

12. 

an  item. 
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(domains)  of  HSI. 

2.3.  When  each  domain  is  “moused- 

over”  the  phrase  identified  on 
the  ‘bumper  sticker’  effectively 
identifies  the  meaning 
(definition  or  function)  of  that 
area. 

13. 

an  item. 

2.4.  Each  domain  definition  and 

example  slide  accurately 
reflects  the  domain. 

14. 

an  item. 

15. 

2.5.  The  page  displaying  all  of  the 

domain  definitions  clearly  and 
accurately  shows  the 
differences  between  each 
domain. 

16. 

an  item. 

17. 

2.6.  Based  on  the  information 

presented  in  this  section,  what 
is  the  goal  of  HSI? 

2.7.  Do  you  have  any 

recommendations  for 
improvement? 

3.  Please  click  on  the  TSA  Scenario  button. 

3.1 .  The  scenario  description  was 

informative. 

18. 

an  item. 

4.  Please  move  on  to  the  Introduction  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen. 

4.1 .  The  task  is  clearly  and 

concisely  stated.  If  not,  please 
explain. 

19. 

an  item. 

4.2.  1  understand  my  goals  and 

objectives  for  this  task.  What 
are  the  goals  and  objectives? 

20.  Choose 

an  item. 

5.  Please  move  on  to  the  Systems  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen. 

5.1  1  understand  the  three  pieces 

of  data  1  was  provided  for  each 
system  (cost,  required 
manpower,  passenger 
throughput).  If  not,  please 
explain. 

21. 

an  item. 

22.  5.2  The  overarching 

objectives  in  choosing  a  system 
were  easy  to  understand. 

Please  identify  your  overarching 
objectives  in  choosing  a 
system. 

23.  Choose 

an  item. 

6.  Please  move  on  to  the  Analysis  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen. 

6.1  1  understand  the  importance  of 

performing  HSI  analysis  to 

24. 

an  item. 
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assist  in  identifying  the  optimal 
solution.  If  not,  please  explain. 

25.  6.2  1  understand  what  will 

be  provided  if  1  purchase  HSI 
analysis.  What  will  HSI  analysis 
provide  to  your  decision 
making? 

26.  Choose 

an  item. 

6.3  Please  identify  what  type(s)  of 

HSI  analysis  you  purchased. 

27.  Choose 

an  item. 

7.  If  you  chose  to  purchase  HSI  Analysis  please  move  on  to  the  Personnel  and  Training  Needs 
Analysis  screen  by  clicking  on  the  forward  arrow  at  the  bottom  of  the  right  screen. 

7.1  Did  you  purchase  Personnel 

and  Training  Needs  Analysis? 
Why? 

28.  Choose 

an  item. 

7.2  The  Personnel  and  Training 

Needs  Analysis  identified  that 
personnel  skills  and  required 
training  are  indirectly 
proportional. 

29.  Choose 

an  item. 

7.3  The  graph  accurately 

represents  the  relationship 
between  the  aptitude  scores 
and  the  required  training. 

30.  Choose 

an  item. 

7.4  1  can  effectively  translate  the 

Personnel  and  Training  Needs 
Analysis  into  my  decision 
making  for  choosing  a  new 
scanning  system.  If  not,  please 
explain. 

31. 

an  item. 

8.  If  you  chose  to  purchase  HSI  analysis  please  move  on  to  the  Throughput  and  Manpower  Analysis 
screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the  screen. 

8.1  Did  you  purchase  Throughput 

and  Manpower  Analysis?  Why? 

32.  Choose 

an  item. 

33.  8.2  1  understand  how 

throughput  and  manpower  will 
affect  the  total  operating  cost.  If 
not,  please  explain. 

34. 

an  item. 

35.  8.3  1  understand  the  data 

represented  on  each  graph 
showing  the  relationship 
between  the  throughput  and 
skill  level  to  meet  the  required 
detection  rate.  If  not,  please 
explain. 

36.  Choose 

an  item. 

37.  8.4  1  understand  that  based 

on  operator  skill  level  1  will  need 
a  different  number  of  systems  to 
achieve  required  throughput.  If 
not,  please  explain. 

38.  Choose 

an  item. 

9.  Please  move  onto  the  Training  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  You  have  reached  the  screen  where  you  will  begin  making  tradeoff  decisions  on  the  different 
domains  in  order  to  accomplish  your  overall  goals.  Please  read  the  information  presented  at  the  top 
of  the  screen  and  choose  Personnel  and  Training  options. 
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9.1.  Enough  information  is 

presented  for  me  to  make  a 
decision  on  the  aptitude  scores 
an  individual  may  need.  If  not, 
what  additional  information 
would  allow  you  to  make  a  more 
informed  decision? 

39.  Choose 

an  item. 

9.2.  Enough  information  is 

presented  for  me  to  make  a 
decision  on  the  amount  of 
training  hours  an  individual  may 
need.  If  not,  what  additional 
information  would  allow  you  to 
make  a  more  informed 
decision  ? 

40. 

an  item. 

10.  Please  move  on  to  the  Selection  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  read  the  information  presented  at  the  top  of  the  screen. 

10.1.  Based  on  the  training  and 

personnel  decisions  you  made 
on  the  previous  screen,  it  was 
difficult  to  determine  the 
appropriate  equipment  to  meet 
overall  scenario  goals. 

41. 

an  item. 

11.  Please  move  on  to  the  Review  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  review  the  selections  you  have  made. 

11.1.  Do  all  of  the  selections  that  you 
previously  choose  appear?  If 
not,  what  is  not  accurately 
represented? 

42. 

an  item. 

11.2.  Did  you  go  back  to  a  screen 

and  make  changes.  If  so,  what 
screen?  What  change  did  you 
make? 

43. 

an  item. 

1 1 .3.  What  Personnel  choice  did  you 
make? 

44. 

an  item. 

11.4.  What  Training  choice  did  you 
make? 

45. 

an  item. 

11.5.  What  System  did  you  chose? 

46. 

an  item. 

12.  Please  move  on  to  the  Results  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  review  results  of  your  tradeoff  decisions 

12.1.  The  resulting  information  was 
what  1  expected.  If  not,  why? 
What  did  you  expect? 

47. 

an  item. 

12.2.  Did  your  system  meet  all  the 
goals  and  objectives? 

48. 

an  item. 

12.3.  If  not,  what  goal(s)  didn’t  you 
meet? 

49. 

an  item. 

12.4.  If  you  purchased  HSI  analysis, 
did  the  information  provided 
assist  you  in  making  a 
decision? 

50.  Choose 

an  item. 

12.5.  With  the  information  provided 

51. 

an  item. 
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would  you  be  able  to  go  back 
and  make  a  better  informed 
decision  if  you  were  to  go 
through  the  scenario  again? 

12.6.  What  different  tradeoffs  would 
you  make  in  the  future? 

13.  Please  move  on  to  the  Solutions  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  corner. 

13.1.  This  screen  accurately 
represents  the  optimized 
solution. 

52. 

an  item. 

13.2.  1  understand  the  difference  in 

the  procurement  and  operating 
costs. 

53.  Choose 

an  item. 

13.3.  Why  is  this  solution  optimal? 

13.4.  Please  provide  any  additional 

comments/recommendations  for 
improvement  on  the  tool  and 
scenario. 

14.  Please  click  on  the  UAS  Scenario  button. 

14.1.  The  scenario  description  was 
informative. 

54. 

an  item. 

15.  Please  proceed  to  the  next  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

15.1  1  understand  the  task 

objectives.  What  are  the 
objectives? 

55.  Choose 

an  item. 

16.  Please  proceed  to  the  Introduction  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

16.1.  1  understand  the  UAS  scenario 

goals.  What  are  the  goals? 

56.  Choose 

an  item. 

16.2.  Acceptable  workload  is  easily 

understood  in  the  context  of  this 
scenario. 

57.  Choose 

an  item. 

17.  Please  proceed  to  the  next  Introduction  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

17.1.  The  three  design  tasks  I’m 

expected  to  perform  within  this 
scenario  are  clearly  identified. 
Please  list  the  tasks. 

58.  Choose 

an  item. 

17.2.  Enough  background  information 
is  presented  to  assist  in  making 
HSI  tradeoffs  for  UAS  control 
station  design  and  operations.  If 
not,  please  explain. 

Choose  an  item. 

18.  Please  proceed  to  the  Screens  screen  by  clicking  on  the  arrow  at  the  bottom  right. 

18.1  1  understand  the  screens 

available  for  inclusion  in  the 
design. 

Choose  an  item. 

18.2  Each  definition  is  easily 
understood  and  clearly 
represents  the  design  element. 

If  not,  please  explain. 

Choose  an  item. 

19.  Please  proceed  to  the  Analysis  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

59.  19.1  1  understand  the 

importance  of  performing  HSI 
analysis  to  assist  in  identifying 

60.  Choose 

an  item. 
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the  optimal  solution.  If  not, 
please  explain. 

19.2  1  understand  what  will  be 

provided  if  1  purchase  HSI 
analysis.  What  will  HSI  analysis 
provide  to  your  decision 
making? 

61. 

an  item. 

1 9.3  Please  identify  what  type(s)  of 
HSI  analysis  you  purchased. 

62.  Choose 

an  item. 

20.  If  you  chose  to  conduct  Task  and  Workload  Analysis  please  proceed  to  the  Task  and  Workload 
Analysis  screens  by  clicking  on  the  forward  arrow  in  the  bottom  right. 

20.1  Did  you  purchase  Task  and 
Workload  Analysis?  Why? 

Choose  an  item. 

63.  20.2  1  understand  the 

definition  of  task  analysis. 

Choose  an  item. 

20.3  Please  identify  the  role  of  the 
cognitive  task  analysis  in 
redesigning  the  UAS  displays. 

21.  Please  proceed  to  the  next  Task  and  Workload  Analysis  screen  by  clicking  on  the  forward  arrow  in 
the  bottom  right. 

21.1  1  understand  the  definition  of 

workload  and  how  maintaining 
situation  awareness  will  affect 
the  workload.  If  not,  please 
explain. 

Choose  an  item. 

22.  Please  proceed  to  the  next  Task  and  Workload  Analysis  screen  by  clicking  on  the  forward  arrow  in 
the  bottom  right. 

22.1  The  mission  critical  tasks  are 
clearly  identified. 

Choose  an  item. 

22.2  Please  list  the  mission  critical 
tasks  that  will  be  major  design 
drivers. 

22.3  1  understand  the  probability  of 

each  primary  task  occurring. 
Please  list  the  order  of 
frequency. 

Choose  an  item. 

22.4  1  can  easily  identify  the 

relationships  between  these 
tasks  and  some  of  the  design 
screens  previously  presented? 

If,  not  please  describe  why. 

Choose  an  item. 

22.5  1  understand  each  primary  task 

that  will  be  performed  while 
operating  a  UAS.  If  not,  please 
identify  which  task(s)  were  not 
clearly  described. 

Choose  an  item. 

22.6  Workload  is  a  function  of  time 
and  effort  needed  to  complete 
the  primary  tasks. 

Choose  an  item. 

23.  Please  proceed  to  the  next  Task  and  Workload  Analysis  screen  by  clicking  on  the  forward  arrow  in 
the  bottom  right. 

23.1  1  understand  that  visual 

workload  is  also  defined  by  the 

Choose  an  item. 
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distance  and  frequency  of 
performing  a  task. 

23.2  How  many  monitor 

configuration  options  are 
available  for  designing  the 
control  station? 

Choose  an  item. 

23.3  The  information  presented  on 

Task  and  Workload  Analysis  will 
assist  in  designing  the  UAS 
displays.  If  not,  why? 

Choose  an  item. 

24.  If  you  chose  to  purchase  Training  Needs  Analysis,  please  proceed  to  the  Training  Needs  Analysis 
screen  by  clicking  on  the  arrow  at  the  bottom  right. 

24.1  Did  you  purchase  Training 

Needs  Analysis?  Why? 

Choose  an  item. 

24.2  This  screen  clearly  identifies 
that  aptitude  score  is  indirectly 
proportional  to  the  required 
training. 

Choose  an  item. 

24.3  The  minimum  amount  of 
training  required  for  each 
operator  is  easily  identified. 

Choose  an  item. 

25.  Please  proceed  to  the  next  Training  Needs  Analysis  screen  by  clicking  on  the  arrow  at  the  bottom 
right. 

25.1  The  graph  accurately  portrays 
the  relationship  between  the 
task  completion  time  and  the 
operator  skill  level. 

Choose  an  item. 

26.  Please  proceed  to  the  Monitor  choices  screen  by  clicking  on  the  arrow  at  the  bottom  right.  If  you 
chose  not  to  purchase  HSI  analysis  this  screen  will  directly  follow  the  Introduction  screens. 

26.1  1  can  easily  choose  a  monitor 

configuration 

Choose  an  item. 

26.2  Please  identify  what  monitor 
configuration  you  chose. 

Choose  an  item. 

27.  Please  proceed  to  the  Design  screen  by  clicking  on  the  arrow  at  the  bottom  right. 

27.1.  The  description  clearly  explains 
how  to  place  the  display  screens 
within  the  monitors. 

Choose  an  item. 

27.2.  Enough  information  is 

presented  for  me  to  optimize 
display  design.  If  not,  please 
explain  why. 

Choose  an  item. 

27.3.  Please  list  the  screens  you 
chose. 

28.  Please  proceed  to  the  Operators  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

28.1.  1  understand  the  differences 

between  each  operator’s 
background  and  skills.  If  not, 
please  explain. 

Choose  an  item. 

28.2.  Enough  information  is 

presented  for  me  to  choose  an 
operator.  If  not,  please  explain 
why. 

Choose  an  item. 

29.  Please  proceed  to  the  Training  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 
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29.1.  Enough  information  is 

presented  for  me  to  make  a 
decision  on  the  amount  of 
training  hours  an  individual  may 
need?  If  not,  what  additional 
information  would  allow  you  to 
make  a  more  informed 
decision  ? 

64. 

an  item. 

30.  Please  proceed  to  the  Review  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

30. 1 .  Do  all  of  the  selections  that  you 
previously  choose  appear?  If 
not,  what  is  not  accurately 
represented? 

65. 

an  item. 

30.2.  Did  you  go  back  to  a  screen 

and  make  changes.  If  so,  what 
screen?  What  change  did  you 
make? 

66.  Choose 

an  item. 

30.3.  Please  identify  what  monitor 
configuration  you  chose. 

Choose  an  item. 

30.4.  What  operator  choice  did  you 
make? 

67.  Choose 

an  item. 

30.5.  What  training  choice  did  you 
make? 

68.  Choose 

an  item. 

31.  Please  proceed  to  the  Results  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

31.1.  The  resulting  information  was 
what  1  expected.  If  not,  why? 
What  did  you  expect? 

69.  Choose 

an  item. 

31.2.  Did  your  system  meet  all  the 
goals  and  objectives? 

70.  Choose 

an  item. 

31.3.  If  not,  what  goal(s)  didn’t  you 
meet? 

71. 

an  item. 

31.4.  If  you  purchased  HSI  analysis, 
did  the  information  provided 
assist  you  in  making  a 
decision? 

72.  Choose 

an  item. 

31.5.  With  the  results  provided  would 
you  be  able  to  go  back  and 
make  a  better  informed  decision 
if  you  were  to  go  through  the 
scenario  again? 

73.  Choose 

an  item. 

31 .6.  What  different  tradeoffs  would 
you  make  in  the  future? 

32.  Please  proceed  to  the  Solutions  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right. 

74.  32.1  This  screen  accurately 

represents  the  optimal 
solutions. 

75.  Choose 

an  item. 

76.  32.2  1  understand  why  all  of 

these  solutions  are  optimal. 

Choose  an  item. 

77.  32.3  Why  is  this  solution 

optimal? 

78.  32.4  Please  provide  any 

additional  recommendations  to 
improve  this  scenario  or  tool. 
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33.  General  Usability 

33.1  The  demo  was  easy  to  use  and 
requires  little  instruction. 

79.  Choose 

an  item. 

80.  33.2  The  graphical 

information  on  the  screen  is 
easy  to  understand. 

81. 

an  item. 

82.  33.3  It  was  easy  to  navigate 

through  the  screens. 

83.  Choose 

an  item. 

84.  33.4  The  textual  information 

was  easy  to  read  and 
understand. 

85.  Choose 

an  item. 

86.  33.5  The  color  scheme  and 

design  is  aesthetically  pleasing. 

87.  Choose 

an  item. 

88.  33.6  The  demo  was 

designed  to  minimize  the 
potential  for  software  errors  to 
occur. 

89.  Choose 

an  item. 

34.  HSI  Understanding 

90.  34.1  1  have  a  more  thorough 

understanding  of  HSI  domains. 

91. 

an  item. 

92.  34.2  1  have  a  better 

understanding  of  HSI  tradeoffs. 

93.  Choose 

an  item. 

94.  34.3  If  1  encountered  a 

different  scenario  1  would  be 
able  to  identify  potential  HSI 
tradeoffs. 

95.  Choose 

an  item. 

34.5  1  am  interested  in  additional 

information  about  HSI  tradeoffs. 

If  not,  why? 

96.  Choose 

an  item. 
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Appendix  H:  Usability  Test  Artifacts  -  HSI  Tradeoff  Tool  User  Test  2 

Human  Systems  Integration  Marketing  Demonstration  Tradeoff  Tool 

Section  1:  Background  Information 

The  information  you  provide  in  this  section  is  for  statistical  purposes  only.  We  ask  that  you  provide  your 
name  and  answer  the  following  background/demographic  questions. 


Name: 

Comments 

5.  Background/Demographics 

96.1.  Do  you  work  in  the  field  of  HSI? 

If  yes,  please  describe 
activities. 

15. 

an  item. 

96.2.  Are  you  familiar  with  the 
domains  of  Air  Force  HSI? 

16. 

an  item. 

96.3.  Are  you  government  or 
contractor? 

17. 

an  item. 

96.4.  Have  you  participated  in 

developing  marketing  tools  or 
materials  in  the  past?  If  yes,  for 
what  industry. 

18. 

an  item. 

Section  2:  Time 

Please  record  your  start  and  end  times.  Start  with  the  “What  is  HSI?”  link  and  continue  to  the  TSA  Scenario. 
Demonstration  Link:  http://www.bahdayton.com/71 1th 


Time  Initiated:  From  scenario  initiation 

19. 

Time  Completed:  Results  screen 

20. 

Section  3:  Usability/Tasks  Walkthrough 

Please  read  the  instructions  below  and  provide  your  feedback  on  the  tasks  as  they  are  completed.  For  this 
section  of  the  questionnaire,  you  will  go  through  an  example  scenario  of  identifying  HSI-related  tradeoffs. 

Please  rate  the  applicable  statements  on  a  5-point  scale  with  1  representing  the  low  or  negative  end  of 
the  scale  and  5  as  the  high  or  positive  end  of  the  scale  and  provide  comments  to  support  your  rating. 


Ratings 

Comments 

6.  Please  click  on  the  “What  is  HSI?”  label  on  the  Home  page. 

2.8.  The  information  is  presented  in 

a  clear  and  concise  manner.  If 
not,  how  would  you  recommend 
we  present  background/basic 

HSI  information? 

21. 

an  item. 
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2.9.  The  definition  of  HSI  accurately 

represents  the  functions 
(domains)  of  HSI. 

22.  Choose 

an  item. 

2.10.  When  each  domain  is  “moused- 
over”  the  phrase  identified  on 
the  ‘bumper  sticker’  effectively 
identifies  the  meaning 
(definition  or  function)  of  that 
area. 

23.  Choose 

an  item. 

2.11.  Based  on  the  information 

presented  on  this  page,  what  is 
the  goal  of  HSI? 

2.12.  Do  you  have  any 

recommendations  for 
improvement? 

7.  Please  click  on  the  TSA  Scenario  button. 

3.2.  The  scenario  description  was 

informative. 

24. 

an  item. 

8.  Please  move  on  to  the  Introduction  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen. 

4.3.  The  task  is  clearly  and 

concisely  stated.  If  not,  please 
explain. 

25. 

an  item. 

4.4.  1  understand  my  goals  and 

objectives  for  this  task.  What 
are  the  goals  and  objectives? 

26.  Choose 

an  item. 

9.  Please  move  on  to  the  Systems  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen. 

5.1 .  1  understand  the  three  pieces  of 

data  1  was  provided  for  each 
system,  (cost,  required 
manpower,  passenger 
throughput).  If  not,  please 
explain. 

27.  Choose 

an  item. 

5.2.  The  overarching  goal  in 

choosing  a  system  was  easy  to 
understand.  Please  identify  your 
overarching  goal  in  choosing  a 
system 

28.  Choose 

an  item. 

10.  Please  move  onto  the  Training  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  You  have  reached  the  screen  where  you  will  begin  making  tradeoff  decisions  on  the  different 
domains  in  order  to  accomplish  your  overall  goals.  Please  read  the  information  presented  at  the  top 
of  the  screen  and  choose  Personnel  and  Training  options. 

9.3.  Enough  information  is 

presented  for  me  to  make  a 
decision  on  the  aptitude  scores 
an  individual  may  need.  If  not, 
what  additional  information 
would  allow  you  to  make  a  more 
informed  decision? 

29.  Choose 

an  item. 

9.4.  Enough  information  is 

presented  for  me  to  make  a 

30.  Choose 

an  item. 
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decision  on  the  amount  of 
training  hours  an  individual  may 
need.  If  not,  what  additional 
information  would  allow  you  to 
make  a  more  informed 
decision  ? 

11.  Please  move  on  to  the  Selection  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  read  the  information  presented  at  the  top  of  the  screen. 

10.2.  Based  on  the  training  and 

personnel  decisions  you  made 
on  the  previous  screen,  it  was 
difficult  to  determine  the 
appropriate  equipment  to  meet 
overall  scenario  goals. 

31. 

an  item. 

12.  Please  move  on  to  the  Review  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  review  the  selections  you  have  made. 

1 1 .6.  Do  all  of  the  selections  that  you 
previously  choose  appear?  If 
not,  what  is  not  accurately 
represented? 

32.  Choose 

an  item. 

11.7.  Did  you  go  back  to  a  screen 

and  make  changes.  If  so,  what 
screen?  What  change  did  you 
make? 

33.  Choose 

an  item. 

1 1 .8.  What  Personnel  choice  did  you 
make? 

34. 

an  item. 

11.9.  What  Training  choice  did  you 
make? 

35.  Choose 

an  item. 

11.10.  What  System  did  you  chose? 

36.  Choose 

an  item. 

13.  Please  move  on  to  the  Results  screen  by  clicking  on  the  forward  arrow  at  the  bottom  right  of  the 
screen.  Please  review  results  of  your  tradeoff  decisions 

12.7.  The  resulting  information  was 
what  1  expected.  If  not,  why? 
What  did  you  expect? 

37.  Choose 

an  item. 

12.8.  Did  your  system  meet  all  the 
goals  and  objectives? 

38.  Choose 

an  item. 

12.9.  If  not,  what  goal(s)  didn’t  you 
meet? 

39.  Choose 

an  item. 

12.10.  With  the  information  provided 
would  you  be  able  to  go  back 
and  make  a  better  informed 
decision  if  you  were  to  go 
through  the  scenario  again? 

40. 

an  item. 

12.11.  What  different  tradeoffs  would 
you  make  in  the  future? 

14.  Please  move  on  to  the  Solutions  screen  by  clicking  on  the  text  located  at  the  bottom  left  corner 
“Good  Job”  or  “Didn’t  do  so  well?”  Link. 

13.5.  This  screen  accurately 
represents  the  optimized 
solution. 

41. 

an  item. 

13.6.  1  understand  the  different  in 

acquisition  costs  and  life  cycle 

42. 

an  item. 
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costs. 

13.7.  Why  is  this  solution  optimal? 

13.8.  Please  provide  any  additional 

comments/recommendations  for 
improvement  on  the  tool  and 
scenario. 

15.  General  Usability 

11.1.  The  demo  was  easy  to  use  and 
requires  little  instruction. 

43. 

an  item. 

1 1 .2.  The  graphical  information  on 
the  screen  is  easy  to 
understand. 

44. 

an  item. 

11.3.  It  was  easy  to  navigate  through 
the  screens. 

45. 

an  item. 

1 1 .4.  The  textual  information  was 
easy  to  read  and  understand. 

46. 

an  item. 

11.5.  The  color  scheme  and  design  is 
aesthetically  pleasing. 

47. 

an  item. 

11.6.  The  demo  was  designed  to 
minimize  the  potential  for 
software  errors  to  occur. 

48. 

an  item. 

16.  HSI  Understanding 

12.1.  1  have  a  more  thorough 

understanding  of  HSI  domains. 

49. 

an  item. 

12.2.  1  have  a  better  understanding  of 

HSI  tradeoffs. 

50.  Choose 

an  item. 

12.3.  If  1  encountered  a  different 
scenario  1  would  be  able  to 
identify  potential  HSI  tradeoffs. 

51. 

an  item. 

12.4.  1  am  interested  in  additional 

information  about  HSI  tradeoffs. 

If  not,  why? 

52.  Choose 

an  item. 
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Appendix  I:  Usability  Test  Artifacts  -  HSI  Tradeoff  Tool  User  Test  Feedback 


The  User  Feedback  spreadsheet  is  available  upon  request. 
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Test  Subject 

2 

3 

5 

6 

Pre-Tests 

1.1  Match  the  following  terms 
with  the  appropriate 
definition. 

A.  Training 

B.  Personnel 

C.  Fluman  Factors  Engineering 

D.  Manpower 

E.  Safety 

F.  Occupational  Health 

C  -  Human  Factors 

Engineering 

-Design  for/around  the 
human 

F  -  Occupational  Health 

-Prevent  long  term  injuries 
and  disabilities 

D  -  Manpower 

-Number  of  spaces  or  billets 

A  -  Training 

-Skill  enhancement 

E  -  Safety 

-Mishap  and  injury  prevention 
B  -  Personnel 

-Types  of  faces 

B  -  Personnel 

-Design  for/around  the 
human 

F  -  Occupational  Health 

-Prevent  long  term  injuries 
and  disabilities 

C  -  Human  Factors 

Engineering 

-Number  of  spaces  or  billets 

A  -  Training 

-Skill  enhancement 

E  -  Safety 

-Mishap  and  injury  prevention 
D  -  Manpower 
-Types  of  faces 

C  -  Human  Factors 

Engineering 

-Design  for/around  the 
human 

F  -  Occupational  Health 

-Prevent  long  term  injuries 
and  disabilities 

B  -  Personnel 

-Number  of  spaces  or  billets 

A  -  Training 

-Skill  enhancement 

E  -  Safety 

-Mishap  and  injury  prevention 
D  -  Manpower 

-Types  of  faces 

C  -  Human  Factors 

Engineering 

-Design  for/around  the 
human 

E  -  Safety 

-Prevent  long  term  injuries 
and  disabilities 

B  -  Personnel 

-Number  of  spaces  or  billets 

A  -  Training 

-Skill  enhancement 

F  -  Occupational  Health 

-Mishap  and  injury  prevention 
D  -  Manpower 

-Types  of  faces 

1.2  HSI  is... 

A.  Human  Systems  Interaction 

B.  Human  Factors  Engineering 

C.  Human  Systems  Integration 

D.  Human  Systems 
Interdependence 

C  -  Human  Systems  Integration 

C  -  Human  Systems  Interaction 

C  -  Human  Systems  Integration 

C  -  Human  Systems  Integration 

1.3  HSI  in  the  Air  Force  has 
domains 


D  -  Five  (5) 


B  -  Seven  (7) 


A  -  Nine  (9) 


A  -  Nine  (9) 


1.4  HSI  identifies 
interdependencies  and 
tradeoffs  among... 

B  -  HSI  domains  and  Systems 
Engineering  functions 

1.5  HSI  helps  to  ensure  that 
the  human  is  considered  on 
par  with  the  hardware  and 
software  during  system  design 
and  development. 

TRUE 

1.6  Performing  HSI  activities 
supports  increased  human 
performance 

FALSE 

1.7  Effective  consideration  of 
HSI  helps  to  reduce  total 
ownership  costs  and  safety 
mishaps. 

TRUE 

C  -  HSI  domains 


Post-Test 


C  -  HSI  domains 

D  -  Cost,  schedule,  and 
performance 

TRUE 

TRUE 

TRUE 

TRUE 

TRUE 

TRUE 

C  -  Human  Factors 

C  -  Human  Factors 

C  -  Human  Factors 

C  -  Human  Factors 

Engineering 

Engineering 

Engineering 

Engineering 

--Design  for/around  the 

-Design  for/around  the 

-Design  for/around  the 

-Design  for/around  the 

human 

human 

human 

human 

F  -  Occupational  Health 

F  -  Occupational  Health 

F  -  Occupational  Health 

F  -  Occupational  Health 

-Prevent  long  term  injuries 

-Prevent  long  term  injuries 

-Prevent  long  term  injuries 

-Prevent  long  term  injuries 

1.1  Match  the  following  terms 

and  disabilities 

and  disabilities 

and  disabilities 

and  disabilities 

with  the  appropriate 

D  -  Manpower 

D  -  Manpower 

B  -  Personnel 

D  -  Manpower 

definition. 

-Number  of  spaces  or  billets 

-Number  of  spaces  or  billets 

-Number  of  spaces  or  billets 

-Number  of  spaces  or  billets 

A.  Training 

A  -  Training 

A  -  Training 

A  -  Training 

A  -  Training 

B.  Personnel 

-Skill  enhancement 

-Skill  enhancement 

-Skill  enhancement 

-Skill  enhancement 

C.  Human  Factors  Engineering 

E  -  Safety 

E  -  Safety 

E  -  Safety 

E  -  Safety 

D.  Manpower 

-Mishap  and  injury  prevention 

-Mishap  and  injury  prevention 

-Mishap  and  injury  prevention 

-Mishap  and  injury  prevention 

E.  Safety 

B  -  Personnel 

B  -  Personnel 

D  -  Manpower 

B  -  Personnel 

F.  Occupational  Health 

-Types  of  faces 

-Types  of  faces 

-Types  of  faces 

-Types  of  faces 

1.2  HSI  is... 

A.  Human  Systems  Interaction 

B.  Human  Factors  Engineering 

C.  Human  Systems  Integration 

D.  Human  Systems 
Interdependence 

C  -  Human  Systems  Integration 

C  -  Human  Systems  Integration 

C  -  Human  Systems  Integration 

C  -  Human  Systems  Integration 

1.3  HSI  in  the  Air  Force  has 

domains 

A  -  Nine  (9) 

A  -  Nine  (9) 

A  -  Nine  (9) 

A  -  Nine  (9) 

1.4  HSI  identifies 
interdependencies  and 

B  -  HSI  domains  and  Systems 

B  -  HSI  domains  and  Systems 

D  -  Cost,  schedule,  and 

tradeoffs  among... 

A  -  HSI  factors  and  cost 

Engineering  functions 

Engineering  functions 

performance 

1.5  HSI  helps  to  ensure  that 
the  human  is  considered  on 
par  with  the  hardware  and 
software  during  system  design 
and  development. 

TRUE 

1.6  Performing  HSI  activities 
supports  increased  human 
performance 

FALSE 

1.7  Effective  consideration  of 
HSI  helps  to  reduce  total 
ownership  costs  and  safety 
mishaps. 

TRUE 

TRUE 

TRUE 

TRUE 

TRUE 

TRUE 

TRUE 

